Added browser tests using wasmtime test artifacts #42

jsoverson · 2023-10-10T22:04:42Z

This PR adds the start of browser-based tests. It adds a playwright configuration to test across Firefox, Chromium, and Webkit and uses test artifacts from the wasmtime project to assert compatibility.

Only one test has been added so far (test-programs/preview1_path_open_read_write). It required some project changes that need review before moving forward.

To run tests:

$ npm install
$ npm run test:build  # clone wasmtime and compile test .wasm
$ npm run build       # compile project typescript
$ npm run test        # run tests

Change Notes:

The initialize() function was updated to only call _initialize() if it's exported. AFAICT _initialize() is optional. This lets all reactor libraries use the same shim init logic without needing the user to inspect the exports.
I had to pull the wasiImport types into a proper interface to keep satisfying TypeScript. This had the side effect of requiring some changes to satisfy types that would have been the source of errors before. ~~Any change where obj[key] = undefined becomes delete obj[key] is due to this.~~ (reverted)
I needed to propagate the rights flags through to the Fd implementations so that fd_fdstat_get could report correct values.
I added a dumb path_unlink_file implementation to Dir to satisfy the test wasm. It deletes the instance from the dir's contents and doesn't do anything more destructive.

jsoverson · 2023-10-10T22:15:45Z

@bjorn3 I hope these changes are again in the right direction. Testing could go any one of a dozen ways, and this seemed like the best cost/value.

bjorn3 · 2023-10-10T22:20:10Z

Using wasmtime's wasi tests is a great idea. Thanks for working on this. FYI I probably won't have time to review this in the next couple of days.

Part of #7

jsoverson · 2023-10-10T22:42:20Z

@bjorn3 np, thanks for being willing and responsive!

bjorn3 · 2023-10-17T15:08:46Z

.gitignore

+/playwright-report/
+/playwright/.cache/
+
+wasmtime/


nit: missing trailing newline

bjorn3 · 2023-10-17T15:14:32Z

src/fs_fd.ts

+
+  path_unlink_file(path: string): number {
+    delete this.dir.contents[path];
+    return wasi.ERRNO_SUCCESS;


Could you add a FIXME to check if the path actually exists and to return an error otherwise. Also is there no test in wasmtime's test suite for that?

bjorn3 · 2023-10-17T15:15:53Z

src/fs_fd.ts

-    if (this.file.readonly) return { ret: wasi.ERRNO_BADF, nwritten };
+    if (
+      this.file.readonly ||
+      (Number(this.fs_rights_base) & wasi.RIGHTS_FD_WRITE) === 0


Would making RIGHTS_FD_WRITE a Bigint work to avoid this cast?

bjorn3 · 2023-10-17T15:16:54Z

src/fs_fd.ts

+      this.file.readonly ||
+      (Number(this.fs_rights_base) & wasi.RIGHTS_FD_WRITE) === 0
+    )
+      return { ret: wasi.ERRNO_BADF, nwritten };


For please use {} for if where the action is not on the same line. That helps prevent issues like the duplicate "goto fail;" that caused a huge security issue in macOS a couple of years back.

bjorn3 · 2023-10-17T15:18:58Z

scripts/build-tests.sh

+WASMTIME_DIR=$SCRIPT_DIR/../wasmtime
+
+if ! [[ -d wasmtime ]]; then
+  git clone --recurse-submodules https://github.com/bytecodealliance/wasmtime.git --depth=1 $WASMTIME_DIR;


Are the submodules actually necessary? And if so are all of them necessary?

bjorn3 · 2023-10-17T15:19:35Z

scripts/build-tests.sh

+WASMTIME_DIR=$SCRIPT_DIR/../wasmtime
+
+if ! [[ -d wasmtime ]]; then
+  git clone --recurse-submodules https://github.com/bytecodealliance/wasmtime.git --depth=1 $WASMTIME_DIR;


Could you pin a specific commit of wasmtime for reproducibility in the future?

bjorn3 · 2023-10-17T15:20:43Z

scripts/build-tests.sh

+
+cargo build --manifest-path $WASMTIME_DIR/crates/test-programs/artifacts/Cargo.toml
+
+TARGET_DIR=$(ls -dt $WASMTIME_DIR/target/debug/build/test-programs-artifacts-* | head -n 1)


Suggested change

TARGET_DIR=$(ls -dt $WASMTIME_DIR/target/debug/build/test-programs-artifacts-* | head -n 1)

TARGET_DIR=$(ls -dt $WASMTIME_DIR/target/debug/build/test-programs-artifacts-*/out | head -n 1)

There is no guarantee that the first one is the one with the OUT_DIR.

bjorn3 · 2023-10-17T15:23:26Z

I'm not entirely sure all fs_rights_base changes are correct. Would you mind pulling out all changes to the actual code except for fs_rights_base into a separate PR? Also could you add CI integration?

bjorn3 · 2023-11-25T19:58:13Z

@jsoverson are you still interested in working on this or would you prefer if I work on splitting it up into several commits and landing it myself?

jsoverson · 2023-11-25T21:36:59Z

@bjorn3 sorry, this fell off my radar during a few hectic weeks. I'll jump on it this coming week.

jsoverson · 2023-12-01T17:56:12Z

Status: I'm going to fork wasmtime to create releases consisting of strictly the test wasm files. Getting CI setup to clone wasmtime to build the test artifacts is excessive.

bjorn3 · 2023-12-01T19:00:37Z

You can use --filter=blob:none when cloning to tell git to only fetch the blobs that are actually necessary. For me that only takes 10s for a total download of 24MB. It can be reduced even further using a sparse checkout, but I think 10s is more than fast enough.

jsoverson · 2023-12-01T19:16:26Z

It's more the overhead of setting up a rust environment to build wasm files that really shouldn't change. It will dramatically extend the duration of CI tests, vs. downloading a tarball of wasm artifacts.

bjorn3 · 2023-12-01T19:22:42Z

I see. How about caching the wasm test files in a github actions cache after they are built. And then only setup a rust toolchain for building if the cache is missing. That should be easier to keep up to date with new wasmtime versions than building locally and checking into a repo.

added browser tests

b719a54

jsoverson force-pushed the browser-testing branch from dbe77a6 to b719a54 Compare October 10, 2023 22:37

bjorn3 reviewed Oct 17, 2023

View reviewed changes

.gitignore

/playwright-report/

/playwright/.cache/

wasmtime/

Copy link

Owner

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing trailing newline

bjorn3 reviewed Oct 17, 2023

View reviewed changes

bjorn3 mentioned this pull request Nov 14, 2023

feature request: Implement both path_remove_directory and path_unlink_file #49

Closed

bjorn3 mentioned this pull request Dec 5, 2023

test: Setup wasi-testsuite with Node.js adapter #56

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added browser tests using wasmtime test artifacts #42

Added browser tests using wasmtime test artifacts #42

jsoverson commented Oct 10, 2023 •

edited

Loading

jsoverson commented Oct 10, 2023

bjorn3 commented Oct 10, 2023

jsoverson commented Oct 10, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 Oct 17, 2023

bjorn3 commented Oct 17, 2023

bjorn3 commented Nov 25, 2023

jsoverson commented Nov 25, 2023

jsoverson commented Dec 1, 2023

bjorn3 commented Dec 1, 2023

jsoverson commented Dec 1, 2023

bjorn3 commented Dec 1, 2023


		cargo build --manifest-path $WASMTIME_DIR/crates/test-programs/artifacts/Cargo.toml

		TARGET_DIR=$(ls -dt $WASMTIME_DIR/target/debug/build/test-programs-artifacts-* \| head -n 1)

Added browser tests using wasmtime test artifacts #42

Are you sure you want to change the base?

Added browser tests using wasmtime test artifacts #42

Conversation

jsoverson commented Oct 10, 2023 • edited Loading

jsoverson commented Oct 10, 2023

bjorn3 commented Oct 10, 2023

jsoverson commented Oct 10, 2023

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 Oct 17, 2023

Choose a reason for hiding this comment

bjorn3 commented Oct 17, 2023

bjorn3 commented Nov 25, 2023

jsoverson commented Nov 25, 2023

jsoverson commented Dec 1, 2023

bjorn3 commented Dec 1, 2023

jsoverson commented Dec 1, 2023

bjorn3 commented Dec 1, 2023

jsoverson commented Oct 10, 2023 •

edited

Loading